<!DOCTYPE html>
<html lang="en" itemscope itemtype="https://schema.org/WebPage">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Vitess / Server Configuration</title>

        <!-- Webfont -->
    <link href='http://fonts.googleapis.com/css?family=Roboto:400,100,100italic,300,300italic,400italic,500,500italic,700,700italic,900,900italic' rel='stylesheet' type='text/css'>
    
    <!--<link rel="shortcut icon" type="image/png" href="/favicon.png">-->

    <!-- Bootstrap -->
    <link href="/libs/bootstrap/css/bootstrap.min.css" rel="stylesheet">

    <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
    <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
    <!--[if lt IE 9]>
      <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
      <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
    <![endif]-->

    <!-- Styles -->
    <link rel="stylesheet" type="text/css" href="/css/site.css" />
    <!-- Font Awesome icons -->
    <link href="/libs/font-awesome-4.4.0/css/font-awesome.min.css"
          rel="stylesheet"
          type="text/css">
    
    <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.2/jquery.min.js"></script>
    <!-- Include all compiled plugins (below), or include individual files as needed -->
    <script src="/libs/bootstrap/js/bootstrap.min.js"></script>


    <!-- metadata -->
    <meta name="og:title" content="Vitess / Server Configuration"/>
    <meta name="og:image" content="/images/vitess_logo_with_border.svg"/>
    <meta name="og:description" content="Vitess is a database clustering system for horizontal scaling of MySQL."/>

    <link rel="icon" href="/images/vitess_logo_icon_size.png" type="image/png">

    
    <script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
  ga('create', 'UA-60219601-1', 'auto');
  ga('send', 'pageview');
</script>

    
  </head>
  <body class="docs" id="top_of_page">

    <span id="toc-depth" data-toc-depth="h2,h3"></span>


    <nav id="common-nav" class="navbar navbar-fixed-top inverse">
  <div class="container">
    <!-- Brand and toggle get grouped for better mobile display -->
    <div class="navbar-header">
      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar-collapse-1">
        <span class="sr-only">Toggle navigation</span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
        <span class="icon-bar"></span>
      </button>
      <a class="navbar-brand" href="/">
        <img class="logo" src="/images/vitess_logo_with_border.svg" alt="Vitess">
      </a>
    </div>

    <!-- Collect the nav links, forms, and other content for toggling -->
    <div class="collapse navbar-collapse" id="navbar-collapse-1">
      <ul class="nav navbar-nav navbar-right" id="standard-menu-links">
        <li><a href="/overview/">Overview</a></li>
        <li><a href="/user-guide/introduction.html">Guides</a></li>
        <li><a href="/reference/vitess-api.html">Reference</a></li>
        <li><a href="http://blog.vitess.io">Blog</a></li>
        <li><a href="https://github.com/youtube/vitess"><i class="fa fa-github"></i> GitHub</a></li>
        <!-- Hide link to blog unless we have actual posts -->
        <!-- <li><a href="/blog/" title="">Blog</a></li> -->
      </ul>
      <ul class="nav nav-stacked mobile-left-nav-menu" id="collapsed-left-menu">
                <li class="submenu">
          <h4 class="arrow-r no-top-margin">Overview</h4>
          <ul style="display: none">
            <li><a href="/overview/">What is Vitess</a></li>
            <li><a href="/overview/scaling-mysql.html">Scaling MySQL with Vitess</a></li>
            <li><a href="/overview/concepts.html">Key Concepts</a></li>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">Getting Started</h4>
          <ul style="display: none">
            <li style="padding-bottom: 0"><a href="/getting-started/">Run Vitess on Kubernetes</a>
              <ul style="display: block">
                <li style="padding-bottom: 0"><a href="/getting-started/docker-build.html">Custom Docker Build</a></li>
              </ul>
            </li>
            <li><a href="/getting-started/local-instance.html">Run Vitess Locally</a></li>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">User Guide</h4>
          <ul style="display: none">
            <li><a href="/user-guide/introduction.html">Introduction</a>
            <li><a href="/user-guide/client-libraries.html">Client Libraries</a>
            <li><a href="/user-guide/backup-and-restore.html">Backing Up Data</a>
            <li><a href="/user-guide/reparenting.html">Reparenting</a></li>
            <li style="padding-bottom: 0"><a href="/user-guide/schema-management.html">Schema Management</a></li>
              <ul style="display: block">
                <li style="padding-bottom: 0"><a href="/user-guide/vschema.html">VSchema Guide</a></li>
                <li style="padding-bottom: 0"><a href="/user-guide/schema-swap.html">Schema Swap (Tutorial)</a></li>
              </ul>
            <li style="padding-bottom: 0"><a href="/user-guide/sharding.html">Sharding</a>
              <ul style="display: block">
                <li><a href="/user-guide/horizontal-sharding-workflow.html">Horizontal Sharding (Tutorial, automated)</a></li>
                <li><a href="/user-guide/horizontal-sharding.html">Horizontal Sharding (Tutorial, manual)</a></li>
                <li><a href="/user-guide/sharding-kubernetes-workflow.html">Sharding in Kubernetes (Tutorial, automated)</a></li>
                <li style="padding-bottom: 0"><a href="/user-guide/sharding-kubernetes.html">Sharding in Kubernetes (Tutorial, manual)</a></li>
              </ul>
            </li>
            <li><a href="/user-guide/vitess-sequences.html">Vitess Sequences</a></li>
            <li><a href="/user-guide/mysql-server-protocol.html">MySQL Server Protocol</a></li>
            <li><a href="/user-guide/vitess-replication.html">Vitess and Replication</a></li>
            <li><a href="/user-guide/update-stream.html">Update Stream</a></li>
            <li><a href="/user-guide/row-based-replication.html">Row Based Replication</a></li>
            <li><a href="/user-guide/topology-service.html">Topology Service</a></li>
            <li><a href="/user-guide/transport-security-model.html">Transport Security Model</a></li>
            <li style="padding-bottom: 0"><a href="/user-guide/launching.html">Launching</a>
              <ul style="display: block">
                <li><a href="/user-guide/scalability-philosophy.html">Scalability Philosophy</a></li>
                <li><a href="/user-guide/production-planning.html">Production Planning</a></li>
                <li><a href="/user-guide/server-configuration.html">Server Configuration</a></li>
                <li><a href="/user-guide/twopc.html">2PC Guide</a></li>
                <li style="padding-bottom: 0"><a href="/user-guide/troubleshooting.html">Troubleshooting</a></li>
              </ul>
            </li>
            <li><a href="/user-guide/upgrading.html">Upgrading</a></li>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">Reference Guides</h4>
          <ul style="display: none">
            <li><a href="/reference/vitess-api.html">Vitess API</a>
            <li><a href="/reference/vtctl.html">vtctl Commands</a>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">Other Resources</h4>
          <ul style="display: none">
            <li><a href="/resources/presentations.html">Presentations</a>
            <li><a href="http://blog.vitess.io/">Blog</a>
            <li><a href="/resources/roadmap.html">Roadmap</a>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">Contributing</h4>
          <ul style="display: none">
            <li><a href="/contributing/">Contributing to Vitess</a>
            <li><a href="/contributing/github-workflow.html">GitHub Workflow</a>
            <li><a href="/contributing/code-reviews.html">Code Reviews</a>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">Internal</h4>
          <ul style="display: none">
            <li><a href="/internal/">Overview</a>
            <li><a href="/internal/release-instructions.html">Release Instructions</a>
            <li><a href="/internal/publish-website.html">Publish Website</a>
          </ul>
        </li>

        <div>
          <form method="get" action="/search/">
            <input id="search-form" name="q" type="text" placeholder="Search">
          </form>
        </div>

        <li><a href="https://github.com/youtube/vitess" id="collapsed-left-menu-repo-link"><i class="fa fa-github"></i> GitHub</a></li>
      </ul>
    </div><!-- /.navbar-collapse -->
  </div><!-- /.container-fluid -->
</nav>

    
<div id="masthead">
  <div class="container">
    <div class="row">
      <div class="col-md-9">
        <h1>Server Configuration</h1>
      </div>
    </div>
  </div>
</div>


<div class="container">
    <div class="row scroll-margin" id="toc-content-row">
    <div class="col-md-2" id="leftCol">
      <ul class="nav nav-stacked mobile-left-nav-menu" id="sidebar">
                <li class="submenu">
          <h4 class="arrow-r no-top-margin">Overview</h4>
          <ul style="display: none">
            <li><a href="/overview/">What is Vitess</a></li>
            <li><a href="/overview/scaling-mysql.html">Scaling MySQL with Vitess</a></li>
            <li><a href="/overview/concepts.html">Key Concepts</a></li>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">Getting Started</h4>
          <ul style="display: none">
            <li style="padding-bottom: 0"><a href="/getting-started/">Run Vitess on Kubernetes</a>
              <ul style="display: block">
                <li style="padding-bottom: 0"><a href="/getting-started/docker-build.html">Custom Docker Build</a></li>
              </ul>
            </li>
            <li><a href="/getting-started/local-instance.html">Run Vitess Locally</a></li>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">User Guide</h4>
          <ul style="display: none">
            <li><a href="/user-guide/introduction.html">Introduction</a>
            <li><a href="/user-guide/client-libraries.html">Client Libraries</a>
            <li><a href="/user-guide/backup-and-restore.html">Backing Up Data</a>
            <li><a href="/user-guide/reparenting.html">Reparenting</a></li>
            <li style="padding-bottom: 0"><a href="/user-guide/schema-management.html">Schema Management</a></li>
              <ul style="display: block">
                <li style="padding-bottom: 0"><a href="/user-guide/vschema.html">VSchema Guide</a></li>
                <li style="padding-bottom: 0"><a href="/user-guide/schema-swap.html">Schema Swap (Tutorial)</a></li>
              </ul>
            <li style="padding-bottom: 0"><a href="/user-guide/sharding.html">Sharding</a>
              <ul style="display: block">
                <li><a href="/user-guide/horizontal-sharding-workflow.html">Horizontal Sharding (Tutorial, automated)</a></li>
                <li><a href="/user-guide/horizontal-sharding.html">Horizontal Sharding (Tutorial, manual)</a></li>
                <li><a href="/user-guide/sharding-kubernetes-workflow.html">Sharding in Kubernetes (Tutorial, automated)</a></li>
                <li style="padding-bottom: 0"><a href="/user-guide/sharding-kubernetes.html">Sharding in Kubernetes (Tutorial, manual)</a></li>
              </ul>
            </li>
            <li><a href="/user-guide/vitess-sequences.html">Vitess Sequences</a></li>
            <li><a href="/user-guide/mysql-server-protocol.html">MySQL Server Protocol</a></li>
            <li><a href="/user-guide/vitess-replication.html">Vitess and Replication</a></li>
            <li><a href="/user-guide/update-stream.html">Update Stream</a></li>
            <li><a href="/user-guide/row-based-replication.html">Row Based Replication</a></li>
            <li><a href="/user-guide/topology-service.html">Topology Service</a></li>
            <li><a href="/user-guide/transport-security-model.html">Transport Security Model</a></li>
            <li style="padding-bottom: 0"><a href="/user-guide/launching.html">Launching</a>
              <ul style="display: block">
                <li><a href="/user-guide/scalability-philosophy.html">Scalability Philosophy</a></li>
                <li><a href="/user-guide/production-planning.html">Production Planning</a></li>
                <li><a href="/user-guide/server-configuration.html">Server Configuration</a></li>
                <li><a href="/user-guide/twopc.html">2PC Guide</a></li>
                <li style="padding-bottom: 0"><a href="/user-guide/troubleshooting.html">Troubleshooting</a></li>
              </ul>
            </li>
            <li><a href="/user-guide/upgrading.html">Upgrading</a></li>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">Reference Guides</h4>
          <ul style="display: none">
            <li><a href="/reference/vitess-api.html">Vitess API</a>
            <li><a href="/reference/vtctl.html">vtctl Commands</a>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">Other Resources</h4>
          <ul style="display: none">
            <li><a href="/resources/presentations.html">Presentations</a>
            <li><a href="http://blog.vitess.io/">Blog</a>
            <li><a href="/resources/roadmap.html">Roadmap</a>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">Contributing</h4>
          <ul style="display: none">
            <li><a href="/contributing/">Contributing to Vitess</a>
            <li><a href="/contributing/github-workflow.html">GitHub Workflow</a>
            <li><a href="/contributing/code-reviews.html">Code Reviews</a>
          </ul>
        </li>
        <li class="submenu">
          <h4 class="arrow-r">Internal</h4>
          <ul style="display: none">
            <li><a href="/internal/">Overview</a>
            <li><a href="/internal/release-instructions.html">Release Instructions</a>
            <li><a href="/internal/publish-website.html">Publish Website</a>
          </ul>
        </li>

        <div>
          <form method="get" action="/search/">
            <input id="search-form" name="q" type="text" placeholder="Search">
          </form>
        </div>

      </ul>
    </div>
    <div class="col-md-3" id="rightCol">
      <div class="nav nav-stacked" id="tocSidebar">
        <div id="toc"></div>
      </div>
    </div>
    <div class="col-md-7" id="centerCol">
      <div id="centerTextCol">
        <h2 id="mysql">MySQL</h2>

<p>Vitess has some requirements on how MySQL should be configured. These will be detailed below.</p>

<p>As a reminder, semi-sync replication is highly recommended. It offers a much better durability story than relying on a disk. This will also let you relax the disk-based durability settings.</p>

<h3 id="versions">Versions</h3>

<p>MySQL versions supported are: MariaDB 10.0, MySQL 5.6 and MySQL 5.7. A number of custom versions based on these exist (Percona, …), Vitess most likely supports them if the version they are based on is supported.</p>

<h3 id="config-files">Config files</h3>

<h4 id="my-cnf">my.cnf</h4>

<p>The main <code class="prettyprint">my.cnf</code> file is generated by
<a href="https://github.com/youtube/vitess/blob/312064b96ac0070d9f8990e57af6f2c0a76a45a9/examples/local/vttablet-up.sh#L66">mysqlctl init</a>
based primarily on
<a href="https://github.com/youtube/vitess/blob/master/config/mycnf/default.cnf">$VTROOT/config/mycnf/default.cnf</a>.
Additional files will be appended to the generated <code class="prettyprint">my.cnf</code> as specified in
a colon-separated list of absolute paths in the <code class="prettyprint">EXTRA_MY_CNF</code> environment
variable. For example, this is typically used to include <a href="https://github.com/youtube/vitess/blob/312064b96ac0070d9f8990e57af6f2c0a76a45a9/examples/local/vttablet-up.sh#L41">flavor-specific
config files</a>.</p>

<p>To customize the <code class="prettyprint">my.cnf</code>, you can either add overrides in an additional
<code class="prettyprint">EXTRA_MY_CNF</code> file, or modify the files in <code class="prettyprint">$VTROOT/config/mycnf</code> before
distributing to your servers. In Kubernetes, you can use a
<a href="http://kubernetes.io/docs/user-guide/configmap/">ConfigMap</a> to overwrite
the entire <code class="prettyprint">$VTROOT/config/mycnf</code> directory with your custom versions,
rather than baking them into a custom container image.</p>

<h4 id="init_db-sql">init_db.sql</h4>

<p>When a new instance is initialized with <code class="prettyprint">mysqlctl init</code> (as opposed to
restarting in a previously initialized data dir with <code class="prettyprint">mysqlctl start</code>),
the <a href="https://github.com/youtube/vitess/blob/master/config/init_db.sql">init_db.sql</a>
file is applied to the server immediatley after executing <code class="prettyprint">mysql_install_db</code>.
By default, this file contains the equivalent of running
<a href="https://dev.mysql.com/doc/refman/5.7/en/mysql-secure-installation.html">mysql_secure_installation</a>,
as well as the necessary tables and grants for Vitess.</p>

<p>If you are running Vitess on top of an existing MySQL instance,
rather than using mysqlctl, you can use this file as a sample of what
grants need to be applied to enable Vitess.</p>

<p>Note that changes to this file will not be reflected in shards that have
already been initialized and had at least one backup taken.
New instances in such shards will automatically restore the latest backup
upon vttablet startup, overwriting the data dir created by mysqlctl.</p>

<h3 id="statement-based-replication-sbr">Statement-based replication (SBR)</h3>

<p>Vitess relies on adding comments to DMLs, which are later parsed on the other end of replication for various post-processing work. The critical ones are:</p>

<ul>
<li>Redirect DMLs to the correct shard during resharding workflow.</li>
<li>Identify which rows have changed for notifying downstream services that wish to subscribe to changes in vitess.</li>
<li>Workflows that allow you to apply schema changes to replicas first, and rotate the masters, which improves uptime.</li>
</ul>

<p>In order to achieve this, Vitess also rewrites all your DMLs to be primary-key based. In a way, this also makes statement based replication almost as efficient as row-based replication (RBR). So, there should be no major loss of performance if you switched to SBR in Vitess.</p>

<p>RBR will eventually be supported by Vitess.</p>

<h3 id="data-types">Data types</h3>

<p>Vitess supports data types at the MySQL 5.5 level. The newer data types like spatial or JSON are not supported yet. Additionally, the TIMESTAMP data type should not be used in a primary key or sharding column. Otherwise, Vitess cannot predict those values correctly and this may result in data corruption.</p>

<h3 id="no-side-effects">No side effects</h3>

<p>Vitess cannot guarantee data consistency if the schema contains constructs with side effects. These are triggers, stored procedures and foreign keys. This is because the resharding workflow and update stream cannot correctly detect what has changed by looking at a statement.</p>

<p>This rule is not strictly enforced. You are allowed to add these things, but at your own risk. As long as you’ve ensured that a certain side-effect will not break Vitess, you can add it to the schema.</p>

<p>Similar guidelines should be used when deciding to bypass Vitess to send statements directly to MySQL.</p>

<p>Vitess also requires you to turn on STRICT_TRANS_TABLES mode. Otherwise, it cannot accurately predict what will be written to the database.</p>

<p>It’s safe to apply backward compatible DDLs directly to MySQL. VTTablets can be configured to periodically check the schema for changes.</p>

<p>There is also work in progress to actively watch the binlog for schema changes. This will likely happen around release 2.1.</p>

<h3 id="autocommit">Autocommit</h3>

<p>MySQL autocommit needs to be turned on.</p>

<p>VTTablet uses connection pools to MySQL. If autocommit was turned off, MySQL will start an implicit transaction (with a point in time snapshot) for each connection and will work very hard at keeping the current view unchanged, which would be counter-productive.</p>

<h3 id="safe-startup">Safe startup</h3>

<p>We recommend to enable <code class="prettyprint">read-only</code> and <code class="prettyprint">skip-slave-start</code> at startup.
The first ensures that writes will not be accepted accidentally,
which could cause split brain or alternate futures.
The second ensures that slaves do not connect to the master before
settings like semisync are initialized by vttablet according to
Vitess-specific logic.</p>

<h3 id="binary-logging">Binary logging</h3>

<p>By default, we enable binary logging everywhere (<code class="prettyprint">log-bin</code>),
including on slaves (<code class="prettyprint">log-slave-updates</code>).
On <em>replica</em> type tablets, this is important to make sure they have the
necessary binlogs in case they are promoted to master.
The slave binlogs are also used to implement Vitess features like
filtered replication (during resharding) and the upcoming update stream
and online schema swap.</p>

<h3 id="global-transaction-id-gtid">Global Transaction ID (GTID)</h3>

<p>Many features of Vitess require a fully GTID-based MySQL replication
topology, including master management, resharding, update stream,
and online schema swap.</p>

<p>For MySQL 5.6+, that means you must use <code class="prettyprint">gtid_mode=ON</code> on all servers.
We also strongly encourage <code class="prettyprint">enforce_gtid_consistency</code>.</p>

<p>Similarly, for MariaDB, you should use <code class="prettyprint">gtid_strict_mode</code> to ensure that
master management operations will fail rather than risk causing data loss
if slaves diverge from the master due to external interference.</p>

<h3 id="monitoring">Monitoring</h3>

<p>In addition to monitoring the Vitess processes, we recommend to monitor MySQL as well. Here is a list of MySQL metrics you should monitor:</p>

<ul>
<li>QPS</li>
<li>Bytes sent/received</li>
<li>Replication lag</li>
<li>Threads running</li>
<li>Innodb buffer cache hit rate</li>
<li>CPU, memory and disk usage. For disk, break into bytes read/written, latencies and IOPS.</li>
</ul>

<h3 id="recap">Recap</h3>

<ul>
<li>2-4 cores</li>
<li>100-300GB data size</li>
<li>Statement based replication (required)</li>
<li>Semi-sync replication

<ul>
<li>rpl_semi_sync_master_timeout is huge (essentially never; there&#39;s no way to actually specify never)</li>
<li>rpl_semi_sync_master_wait_no_slave = 1</li>
<li>sync_binlog=0</li>
<li>innodb_flush_log_at_trx_commit=2</li>
</ul></li>
<li>STRICT_TRANS_TABLES</li>
<li>auto-commit ON (required)</li>
<li>Additional parameters as mentioned in above sections.</li>
</ul>

<h2 id="vitess-servers">Vitess servers</h2>

<p>Vitess servers are written in Go. There are a few Vitess-specific knobs that apply to all servers.</p>

<h3 id="go-version">Go version</h3>

<p>Go, being a young language, tends to add major improvements over each version.
So, the latest Go version is almost always recommended.
Note that the latest Go version may be higher than the minimum version we require for compiling the binaries (see <a href="/getting-started/#prerequisites">&quot;Prerequisites&quot; section in the Getting Started guide</a>).</p>

<h3 id="gomaxprocs">GOMAXPROCS</h3>

<p>You typically don’t have to set this environment variable. The default Go runtime will try to use as much CPU as necessary. However, if you want to force a Go server to not exceed a certain CPU limit, setting GOMAXPROCS to that value will work in most situations.</p>

<h3 id="gogc">GOGC</h3>

<p>The default value for this variable is 100. Which means that garbage is collected every time memory doubles from the baseline (100% growth). You typically don’t have to change this value either. However, if you care about tail latency, increasing this value will help you in that area, but at the cost of increased memory usage.</p>

<h3 id="logging">Logging</h3>

<p>Vitess servers write to log files, and they are rotated when they reach a maximum size. It’s recommended that you run at INFO level logging. The information printed in the log files come in handy for troubleshooting. You can limit the disk usage by running cron jobs that periodically purge or archive them.</p>

<h3 id="grpc">gRPC</h3>

<p>Vitess uses gRPC for communication between client and Vitess, and between Vitess
servers. By default, Vitess does not use SSL.</p>

<p>Also, even without using SSL, we allow the use of an application-provided
CallerID object. It allows unsecure but easy to use authorization using Table
ACLs.</p>

<p>See the
<a href="http://vitess.io/user-guide/transport-security-model.html">Transport Security Model document</a>
for more information on how to setup both of these features, and what command
line parameters exist.</p>

<h2 id="topology-service-configuration">Topology Service configuration</h2>

<p>Vttablet, vtgate, vtctld need the right command line parameters to find the
Topology Server. First the <em>topo_implementation</em> flag needs to be set to one of
<em>zk2</em>, <em>etcd2</em>, or <em>consul</em>. Then they&#39;re all configured as follows:</p>

<ul>
<li>The <em>topo_global_server_address</em> contains the server address / addresses of
the global topology server.</li>
<li>The <em>topo_global_root</em> contains the directory / path to use.</li>
</ul>

<p>Note that the local cell for the tablet must exist and be configured properly in
the Topology Service for vttablet to start. Local cells are configured inside
the topo server, by using the <code class="prettyprint">vtctl AddCellInfo</code> command. See
the <a href="/user-guide/topology-service.html">Topology Service</a> documentation for more
information.</p>

<h2 id="vttablet">VTTablet</h2>

<!-- TODO: auto-generate a different doc from code for the command line parameters, and link from here. -->

<p>VTTablet has a large number of command line options. Some important ones will be covered here. In terms of provisioning these are the recommended values</p>

<ul>
<li>2-4 cores (in proportion to MySQL cores)</li>
<li>2-4 GB RAM</li>
</ul>

<h3 id="directory-configuration">Directory Configuration</h3>

<p>vttablet supports a number of command line options and environment variables
to facilitate its setup.</p>

<p>The <strong>VTDATAROOT</strong> environment variable specifies the toplevel directory for all
data files. If not set, it defaults to <code class="prettyprint">/vt</code>.</p>

<p>By default, a vttablet will use a subdirectory in <strong>VTDATAROOT</strong> named
<code class="prettyprint">vt_NNNNNNNNNN</code> where <code class="prettyprint">NNNNNNNNNN</code> is the tablet id. The <strong>tablet_dir</strong>
command-line parameter allows overriding this relative path. This is useful in
containers where the filesystem only contains one vttablet, in order to have a
fixed root directory.</p>

<p>When starting up and using <code class="prettyprint">mysqlctl</code> to manage MySQL, the MySQL files will be
in subdirectories of the tablet root. For instance, <code class="prettyprint">bin-logs</code> for the binary
logs, <code class="prettyprint">data</code> for the data files, and <code class="prettyprint">relay-logs</code> for the relay logs.</p>

<p>It is possible to host different parts of a MySQL server files on different
partitions. For instance, the data file may reside in flash, while the bin logs
and relay logs are on spindle. To achieve this, create a symlink from
<code class="prettyprint">$VTDATAROOT/&lt;dir name&gt;</code> to the proper location on disk. When MySQL is
configured by mysqlctl, it will realize this directory exists, and use it for the
files it would otherwise have put in the tablet directory. For instance, to host
the binlogs in <code class="prettyprint">/mnt/bin-logs</code>:</p>

<ul>
<li><p>Create a symlink from <code class="prettyprint">$VTDATAROOT/bin-logs</code> to <code class="prettyprint">/mnt/bin-logs</code>.</p></li>
<li><p>When starting up a tablet:</p>

<ul>
<li><code class="prettyprint">/mnt/bin-logs/vt_NNNNNNNNNN</code> will be created.</li>
<li><code class="prettyprint">$VTDATAROOT/vt_NNNNNNNNNN/bin-logs</code> will be a symlink to
<code class="prettyprint">/mnt/bin-logs/vt_NNNNNNNNNN</code></li>
</ul></li>
</ul>

<h3 id="initialization">Initialization</h3>

<ul>
<li>Init_keyspace, init_shard, init_tablet_type: These parameters should be set at startup with the keyspace / shard / tablet type to start the tablet as. Note ‘master’ is not allowed here, instead use ‘replica’, as the tablet when starting will figure out if it is the master (this way, all replica tablets start with the same command line parameters, independently of which one is the master).</li>
</ul>

<h3 id="query-server-parameters">Query server parameters</h3>

<!-- TODO: Change this section to link to auto-generated descriptions. -->

<ul>
<li><strong>queryserver-config-pool-size</strong>: This value should typically be set to the max number of simultaneous queries you want MySQL to run. This should typically be around 2-3x the number of allocated CPUs. Around 4-16. There is not much harm in going higher with this value, but you may see no additional benefits.</li>
<li><strong>queryserver-config-stream-pool-size</strong>: This value is relevant only if you plan to run streaming queries against the database. It’s recommended that you use rdonly instances for such streaming queries. This value depends on how many simultaneous streaming queries you plan to run. Typical values are in the low 100s.</li>
<li><strong>queryserver-config-transaction-cap</strong>: This value should be set to how many concurrent transactions you wish to allow. This should be a function of transaction QPS and transaction length. Typical values are in the low 100s.</li>
<li><strong>queryserver-config-query-timeout</strong>: This value should be set to the upper limit you’re willing to allow a query to run before it’s deemed too expensive or detrimental to the rest of the system. VTTablet will kill any query that exceeds this timeout. This value is usually around 15-30s.</li>
<li><strong>queryserver-config-transaction-timeout</strong>: This value is meant to protect the situation where a client has crashed without completing a transaction. Typical value for this timeout is 30s.</li>
<li><strong>queryserver-config-max-result-size</strong>: This parameter prevents the OLTP application from accidentally requesting too many rows. If the result exceeds the specified number of rows, VTTablet returns an error. The default value is 10,000.</li>
</ul>

<h3 id="db-config-parameters">DB config parameters</h3>

<p>VTTablet requires multiple user credentials to perform its tasks. Since it&#39;s required to run on the same machine as MySQL, it’s most beneficial to use the more efficient unix socket connections.</p>

<p><strong>app</strong> credentials are for serving app queries:</p>

<ul>
<li><strong>db-config-app-unixsocket</strong>: MySQL socket name to connect to.</li>
<li><strong>db-config-app-uname</strong>: App username.</li>
<li><strong>db-config-app-pass</strong>: Password for the app username. If you need a more secure way of managing and supplying passwords, VTTablet does allow you to plug into a &quot;password server&quot; that can securely supply and refresh usernames and passwords. Please contact the Vitess team for help if you’d like to write such a custom plugin.</li>
<li><strong>db-config-app-charset</strong>: The only supported character set is utf8. Vitess still works with latin1, but it’s getting deprecated.</li>
</ul>

<p><strong>dba</strong> credentials will be used for housekeeping work like loading the schema or killing runaway queries:</p>

<ul>
<li><strong>db-config-dba-unixsocket</strong></li>
<li><strong>db-config-dba-uname</strong></li>
<li><strong>db-config-dba-pass</strong></li>
<li><strong>db-config-dba-charset</strong></li>
</ul>

<p><strong>repl</strong> credentials are for managing replication. Since repl connections can be used across machines, you can optionally turn on encryption:</p>

<ul>
<li><strong>db-config-repl-uname</strong></li>
<li><strong>db-config-repl-pass</strong></li>
<li><strong>db-config-repl-charset</strong></li>
<li><strong>db-config-repl-flags</strong>: If you want to enable SSL, this must be set to 2048.</li>
<li><strong>db-config-repl-ssl-ca</strong></li>
<li><strong>db-config-repl-ssl-cert</strong></li>
<li><strong>db-config-repl-ssl-key</strong></li>
</ul>

<p><strong>filtered</strong> credentials are for performing resharding:</p>

<ul>
<li><strong>db-config-filtered-unixsocket</strong></li>
<li><strong>db-config-filtered-uname</strong></li>
<li><strong>db-config-filtered-pass</strong></li>
<li><strong>db-config-filtered-charset</strong></li>
</ul>

<h3 id="monitoring">Monitoring</h3>

<p>VTTablet exports a wealth of real-time information about itself. This section will explain the essential ones:</p>

<h4 id="debug-status">/debug/status</h4>

<p>This page has a variety of human-readable information about the current VTTablet. You can look at this page to get a general overview of what’s going on. It also has links to various other diagnostic URLs below.</p>

<h4 id="debug-vars">/debug/vars</h4>

<p>This is the most important source of information for monitoring. There are other URLs below that can be used to further drill down.</p>

<h5 id="queries-as-described-in-debug-vars-section">Queries (as described in /debug/vars section)</h5>

<p>Vitess has a structured way of exporting certain performance stats. The most common one is the Histogram structure, which is used by Queries:</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">  &quot;Queries&quot;: {
    &quot;Histograms&quot;: {
      &quot;PASS_SELECT&quot;: {
        &quot;1000000&quot;: 1138196,
        &quot;10000000&quot;: 1138313,
        &quot;100000000&quot;: 1138342,
        &quot;1000000000&quot;: 1138342,
        &quot;10000000000&quot;: 1138342,
        &quot;500000&quot;: 1133195,
        &quot;5000000&quot;: 1138277,
        &quot;50000000&quot;: 1138342,
        &quot;500000000&quot;: 1138342,
        &quot;5000000000&quot;: 1138342,
        &quot;Count&quot;: 1138342,
        &quot;Time&quot;: 387710449887,
        &quot;inf&quot;: 1138342
      }
    },
    &quot;TotalCount&quot;: 1138342,
    &quot;TotalTime&quot;: 387710449887
  },
</code></pre></div>
<p>The histograms are broken out into query categories. In the above case, &quot;PASS_SELECT&quot; is the only category.  An entry like <code class="prettyprint">&quot;500000&quot;: 1133195</code> means that <code class="prettyprint">1133195</code> queries took under <code class="prettyprint">500000</code> nanoseconds to execute.</p>

<p>Queries.Histograms.PASS_SELECT.Count is the total count in the PASS_SELECT category.</p>

<p>Queries.Histograms.PASS_SELECT.Time is the total time in the PASS_SELECT category.</p>

<p>Queries.TotalCount is the total count across all categories.</p>

<p>Queries.TotalTime is the total time across all categories.</p>

<p>There are other Histogram variables described below, and they will always have the same structure.</p>

<p>Use this variable to track:</p>

<ul>
<li>QPS</li>
<li>Latency</li>
<li>Per-category QPS. For replicas, the only category will be PASS_SELECT, but there will be more for masters.</li>
<li>Per-category latency</li>
<li>Per-category tail latency</li>
</ul>

<h5 id="results">Results</h5>
<div class="highlight"><pre><code class="language-text" data-lang="text">  &quot;Results&quot;: {
    &quot;0&quot;: 0,
    &quot;1&quot;: 0,
    &quot;10&quot;: 1138326,
    &quot;100&quot;: 1138326,
    &quot;1000&quot;: 1138342,
    &quot;10000&quot;: 1138342,
    &quot;5&quot;: 1138326,
    &quot;50&quot;: 1138326,
    &quot;500&quot;: 1138342,
    &quot;5000&quot;: 1138342,
    &quot;Count&quot;: 1138342,
    &quot;Total&quot;: 1140438,
    &quot;inf&quot;: 1138342
  }
</code></pre></div>
<p>Results is a simple histogram with no timing info. It gives you a histogram view of the number of rows returned per query.</p>

<h5 id="mysql">Mysql</h5>

<p>Mysql is a histogram variable like Queries, except that it reports MySQL execution times. The categories are &quot;Exec&quot; and “ExecStream”.</p>

<p>In the past, the exec time difference between VTTablet and MySQL used to be substantial. With the newer versions of Go, the VTTablet exec time has been predominantly been equal to the mysql exec time, conn pool wait time and consolidations waits. In other words, this variable has not shown much value recently. However, it’s good to track this variable initially, until it’s determined that there are no other factors causing a big difference between MySQL performance and VTTablet performance.</p>

<h5 id="transactions">Transactions</h5>

<p>Transactions is a histogram variable that tracks transactions. The categories are &quot;Completed&quot; and “Aborted”.</p>

<h5 id="waits">Waits</h5>

<p>Waits is a histogram variable that tracks various waits in the system. Right now, the only category is &quot;Consolidations&quot;. A consolidation happens when one query waits for the results of an identical query already executing, thereby saving the database from performing duplicate work.</p>

<p>This variable used to report connection pool waits, but a refactor moved those variables out into the pool related vars.</p>

<h5 id="errors">Errors</h5>
<div class="highlight"><pre><code class="language-text" data-lang="text">  &quot;Errors&quot;: {
    &quot;Deadlock&quot;: 0,
    &quot;Fail&quot;: 1,
    &quot;NotInTx&quot;: 0,
    &quot;TxPoolFull&quot;: 0
  },
</code></pre></div>
<p>Errors are reported under different categories. It’s beneficial to track each category separately as it will be more helpful for troubleshooting. Right now, there are four categories. The category list may vary as Vitess evolves.</p>

<p>Plotting errors/query can sometimes be useful for troubleshooting.</p>

<p>VTTablet also exports an InfoErrors variable that tracks inconsequential errors that don’t signify any kind of problem with the system. For example, a dup key on insert is considered normal because apps tend to use that error to instead update an existing row. So, no monitoring is needed for that variable.</p>

<h5 id="internalerrors">InternalErrors</h5>
<div class="highlight"><pre><code class="language-text" data-lang="text">  &quot;InternalErrors&quot;: {
    &quot;HungQuery&quot;: 0,
    &quot;Invalidation&quot;: 0,
    &quot;MemcacheStats&quot;: 0,
    &quot;Mismatch&quot;: 0,
    &quot;Panic&quot;: 0,
    &quot;Schema&quot;: 0,
    &quot;StrayTransactions&quot;: 0,
    &quot;Task&quot;: 0
  },
</code></pre></div>
<p>An internal error is an unexpected situation in code that may possibly point to a bug. Such errors may not cause outages, but even a single error needs be escalated for root cause analysis.</p>

<h5 id="kills">Kills</h5>
<div class="highlight"><pre><code class="language-text" data-lang="text">  &quot;Kills&quot;: {
    &quot;Queries&quot;: 2,
    &quot;Transactions&quot;: 0
  },
</code></pre></div>
<p>Kills reports the queries and transactions killed by VTTablet due to timeout. It’s a very important variable to look at during outages.</p>

<h5 id="transactionpool">TransactionPool*</h5>

<p>There are a few variables with the above prefix:</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">  &quot;TransactionPoolAvailable&quot;: 300,
  &quot;TransactionPoolCapacity&quot;: 300,
  &quot;TransactionPoolIdleTimeout&quot;: 600000000000,
  &quot;TransactionPoolMaxCap&quot;: 300,
  &quot;TransactionPoolTimeout&quot;: 30000000000,
  &quot;TransactionPoolWaitCount&quot;: 0,
  &quot;TransactionPoolWaitTime&quot;: 0,
</code></pre></div>
<ul>
<li>WaitCount will give you how often the transaction pool gets full that causes new transactions to wait.</li>
<li>WaitTime/WaitCount will tell you the average wait time.</li>
<li>Available is a gauge that tells you the number of available connections in the pool in real-time. Capacity-Available is the number of connections in use. Note that this number could be misleading if the traffic is spiky.</li>
</ul>

<h5 id="other-pool-variables">Other Pool variables</h5>

<p>Just like TransactionPool, there are variables for other pools:</p>

<ul>
<li>ConnPool: This is the pool used for read traffic.</li>
<li>StreamConnPool: This is the pool used for streaming queries.</li>
</ul>

<p>There are other internal pools used by VTTablet that are not very consequential.</p>

<h5 id="tableaclallowed-tableacldenied-tableaclpseudodenied">TableACLAllowed, TableACLDenied, TableACLPseudoDenied</h5>

<p>The above three variables table acl stats broken out by table, plan and user.</p>

<h5 id="querycachesize">QueryCacheSize</h5>

<p>If the application does not make good use of bind variables, this value would reach the QueryCacheCapacity. If so, inspecting the current query cache will give you a clue about where the misuse is happening.</p>

<h5 id="querycounts-queryerrorcounts-queryrowcounts-querytimesns">QueryCounts, QueryErrorCounts, QueryRowCounts, QueryTimesNs</h5>

<p>These variables are another multi-dimensional view of Queries. They have a lot more data than Queries because they’re broken out into tables as well as plan. This is a priceless source of information when it comes to troubleshooting. If an outage is related to rogue queries, the graphs plotted from these vars will immediately show the table on which such queries are run. After that, a quick look at the detailed query stats will most likely identify the culprit.</p>

<h5 id="usertablequerycount-usertablequerytimesns-usertransactioncount-usertransactiontimesns">UserTableQueryCount, UserTableQueryTimesNs, UserTransactionCount, UserTransactionTimesNs</h5>

<p>These variables are yet another view of Queries, but broken out by user, table and plan. If you have well-compartmentalized app users, this is another priceless way of identifying a rogue &quot;user app&quot; that could be misbehaving.</p>

<h5 id="datafree-datalength-indexlength-tablerows">DataFree, DataLength, IndexLength, TableRows</h5>

<p>These variables are updated periodically from information_schema.tables. They represent statistical information as reported by MySQL about each table. They can be used for planning purposes, or to track unusual changes in table stats.</p>

<ul>
<li>DataFree represents data_free</li>
<li>DataLength represents data_length</li>
<li>IndexLength represents index_length</li>
<li>TableRows represents table_rows</li>
</ul>

<h4 id="debug-health">/debug/health</h4>

<p>This URL prints out a simple &quot;ok&quot; or “not ok” string that can be used to check if the server is healthy. The health check makes sure mysqld connections work, and replication is configured (though not necessarily running) if not master.</p>

<h4 id="queryz-debug-query_stats-debug-query_plans-streamqueryz">/queryz, /debug/query_stats, /debug/query_plans, /streamqueryz</h4>

<ul>
<li>/debug/query_stats is a JSON view of the per-query stats. This information is pulled in real-time from the query cache. The per-table stats in /debug/vars are a roll-up of this information.</li>
<li>/queryz is a human-readable version of /debug/query_stats. If a graph shows a table as a possible source of problems, this is the next place to look at to see if a specific query is the root cause.</li>
<li>/debug/query_plans is a more static view of the query cache. It just shows how VTTablet will process or rewrite the input query.</li>
<li>/streamqueryz lists the currently running streaming queries. You have the option to kill any of them from this page.</li>
</ul>

<h4 id="querylogz-debug-querylog-txlogz-debug-txlog">/querylogz, /debug/querylog, /txlogz, /debug/txlog</h4>

<ul>
<li>/debug/querylog is a never-ending stream of currently executing queries with verbose information about each query. This URL can generate a lot of data because it streams every query processed by VTTablet. The details are as per this function: <a href="https://github.com/youtube/vitess/blob/master/go/vt/tabletserver/logstats.go#L202">https://github.com/youtube/vitess/blob/master/go/vt/tabletserver/logstats.go#L202</a></li>
<li>/querylogz is a limited human readable version of /debug/querylog. It prints the next 300 queries by default. The limit can be specified with a limit=N parameter on the URL.</li>
<li>/txlogz is like /querylogz, but for transactions.</li>
<li>/debug/txlog is the JSON counterpart to /txlogz.</li>
</ul>

<h4 id="consolidations">/consolidations</h4>

<p>This URL has an MRU list of consolidations. This is a way of identifying if multiple clients are spamming the same query to a server.</p>

<h4 id="schemaz-debug-schema">/schemaz, /debug/schema</h4>

<ul>
<li>/schemaz shows the schema info loaded by VTTablet.</li>
<li>/debug/schema is the JSON version of /schemaz.</li>
</ul>

<h4 id="debug-query_rules">/debug/query_rules</h4>

<p>This URL displays the currently active query blacklist rules.</p>

<h4 id="debug-health">/debug/health</h4>

<p>This URL prints out a simple &quot;ok&quot; or “not ok” string that can be used to check if the server is healthy.</p>

<h3 id="alerting">Alerting</h3>

<p>Alerting is built on top of the variables you monitor. Before setting up alerts, you should get some baseline stats and variance, and then you can build meaningful alerting rules. You can use the following list as a guideline to build your own:</p>

<ul>
<li>Query latency among all vttablets</li>
<li>Per keyspace latency</li>
<li>Errors/query</li>
<li>Memory usage</li>
<li>Unhealthy for too long</li>
<li>Too many vttablets down</li>
<li>Health has been flapping</li>
<li>Transaction pool full error rate</li>
<li>Any internal error</li>
<li>Traffic out of balance among replicas</li>
<li>Qps/core too high</li>
</ul>

<h2 id="vtgate">VTGate</h2>

<p>A typical VTGate should be provisioned as follows.</p>

<ul>
<li>2-4 cores</li>
<li>2-4 GB RAM</li>
</ul>

<p>Since VTGate is stateless, you can scale it linearly by just adding more servers as needed. Beyond the recommended values, it’s better to add more VTGates than giving more resources to existing servers, as recommended in the philosophy section.</p>

<p>Load-balancer in front of vtgate to scale up (not covered by Vitess). Stateless, can use the health URL for health check.</p>

<h3 id="parameters">Parameters</h3>

<ul>
<li><strong>cells_to_watch</strong>: which cell vtgate is in and will monitor tablets from. Cross-cell master access needs multiple cells here.</li>
<li><strong>tablet_types_to_wait</strong>: VTGate waits for at least one serving tablet per tablet type specified here during startup, before listening to the serving port. So VTGate does not serve error. It should match the available tablet types VTGate connects to (master, replica, rdonly).</li>
<li><strong>discovery_low_replication_lag</strong>: when replication lags of all VTTablet in a particular shard and tablet type are less than or equal the flag (in seconds), VTGate does not filter them by replication lag and uses all to balance traffic.</li>
<li><strong>degraded_threshold (30s)</strong>: a tablet will publish itself as degraded if replication lag exceeds this threshold. This will cause VTGates to choose more up-to-date servers over this one. If all servers are degraded, VTGate resorts to serving from all of them.</li>
<li><strong>unhealthy_threshold (2h)</strong>: a tablet will publish itself as unhealthy if replication lag exceeds this threshold.</li>
<li><strong>transaction_mode (multi)</strong>: <code class="prettyprint">single</code>: disallow multi-db transactions, <code class="prettyprint">multi</code>: allow multi-db transactions with best effort commit, <code class="prettyprint">twopc</code>: allow multi-db transactions with 2pc commit.</li>
<li><strong>normalize_queries (false)</strong>: Turning this flag on will cause vtgate to rewrite queries with bind vars. This is beneficial if the app doesn&#39;t itself send normalized queries.</li>
</ul>

<h3 id="monitoring">Monitoring</h3>

<h4 id="debug-status">/debug/status</h4>

<p>This is the landing page for a VTGate, which can gives you a status on how a particular server is doing. Of particular interest there is the list of tablets this vtgate process is connected to, as this is the list of tablets that can potentially serve queries.</p>

<h4 id="debug-vars">/debug/vars</h4>

<h5 id="vtgateapi">VTGateApi</h5>

<p>This is the main histogram variable to track for vtgates. It gives you a break up of all queries by command, keyspace, and type.</p>

<h5 id="healthcheckconnections">HealthcheckConnections</h5>

<p>It shows the number of tablet connections for query/healthcheck per keyspace, shard, and tablet type.</p>

<h4 id="debug-query_plans">/debug/query_plans</h4>

<p>This URL gives you all the query plans for queries going through VTGate.</p>

<h4 id="debug-vschema">/debug/vschema</h4>

<p>This URL shows the vschema as loaded by VTGate.</p>

<h3 id="alerting">Alerting</h3>

<p>For VTGate, here’s a list of possible variables to alert on:</p>

<ul>
<li>Error rate</li>
<li>Error/query rate</li>
<li>Error/query/tablet-type rate</li>
<li>VTGate serving graph is stale by x minutes (lock server is down)</li>
<li>Qps/core</li>
<li>Latency</li>
</ul>

<!-- TODO: Add vtctld section. -->

<h2 id="external-processes">External processes</h2>

<p>Things that need to be configured:</p>

<h3 id="periodic-backup-configuration">Periodic backup configuration</h3>

<p>We recommend to take backups regularly e.g. you should set up a cron job for it. See our recommendations at  <a href="http://vitess.io/user-guide/backup-and-restore.html#backup-frequency">http://vitess.io/user-guide/backup-and-restore.html#backup-frequency</a>.</p>

<h3 id="logs-archiver-purger">Logs archiver/purger</h3>

<p>You will need to run some cron jobs to archive or purge log files periodically.</p>

<h3 id="orchestrator">Orchestrator</h3>

<p><a href="https://github.com/github/orchestrator">Orchestrator</a> is a tool for
managing MySQL replication topologies, including automated failover.
It can detect master failure and initiate a recovery in a matter of seconds.</p>

<p>For the most part, Vitess is agnostic to the actions of Orchestrator,
which operates below Vitess at the MySQL level. That means you can
pretty much
<a href="https://github.com/github/orchestrator/wiki/Orchestrator-Manual">set up Orchestrator</a>
in the normal way, with just a few additions as described below.</p>

<p>For the <a href="/getting-started/">Kubernetes example</a>, we provide a
<a href="https://github.com/youtube/vitess/blob/master/examples/kubernetes/orchestrator-up.sh">sample script</a>
to launch Orchestrator for you with these settings applied.</p>

<h4 id="orchestrator-configuration">Orchestrator configuration</h4>

<p>Orchestrator needs to know some things from the Vitess side,
like the tablet aliases and whether semisync is enforced
(with async fallback disabled).
We pass this information by telling Orchestrator to execute certain
queries that return local metadata from a non-replicated table,
as seen in our sample
<a href="https://github.com/youtube/vitess/blob/master/docker/orchestrator/orchestrator.conf.json">orchestrator.conf.json</a>:</p>
<div class="highlight"><pre><code class="language-json" data-lang="json">  <span class="s2">&quot;DetectClusterAliasQuery&quot;</span><span class="err">:</span> <span class="s2">&quot;SELECT value FROM _vt.local_metadata WHERE name=&#39;ClusterAlias&#39;&quot;</span><span class="err">,</span>
  <span class="s2">&quot;DetectInstanceAliasQuery&quot;</span><span class="err">:</span> <span class="s2">&quot;SELECT value FROM _vt.local_metadata WHERE name=&#39;Alias&#39;&quot;</span><span class="err">,</span>
  <span class="s2">&quot;DetectPromotionRuleQuery&quot;</span><span class="err">:</span> <span class="s2">&quot;SELECT value FROM _vt.local_metadata WHERE name=&#39;PromotionRule&#39;&quot;</span><span class="err">,</span>
  <span class="s2">&quot;DetectSemiSyncEnforcedQuery&quot;</span><span class="err">:</span> <span class="s2">&quot;SELECT @@global.rpl_semi_sync_master_wait_no_slave AND @@global.rpl_semi_sync_master_timeout &gt; 1000000&quot;</span><span class="err">,</span>
</code></pre></div>
<p>There is also one thing that Vitess needs to know from Orchestrator,
which is the identity of the master for each shard, if a failover occurs.</p>

<p>From our experience at YouTube, we believe that this signal is too critical
for data integrity to rely on bottom-up detection such as asking each MySQL
if it thinks it&#39;s the master. Instead, we rely on Orchestrator to be the
source of truth, and expect it to send a top-down signal to Vitess.</p>

<p>This signal is sent by ensuring the Orchestrator server has access to
<code class="prettyprint">vtctlclient</code>, which it then uses to send an RPC to vtctld, informing
Vitess of the change in mastership via the
<a href="/reference/vtctl.html#tabletexternallyreparented">TabletExternallyReparented</a>
command.</p>
<div class="highlight"><pre><code class="language-json" data-lang="json">  <span class="s2">&quot;PostMasterFailoverProcesses&quot;</span><span class="err">:</span> <span class="p">[</span>
    <span class="s2">&quot;echo &#39;Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Promoted: {successorHost}:{successorPort}&#39; &gt;&gt; /tmp/recovery.log&quot;</span><span class="p">,</span>
    <span class="s2">&quot;vtctlclient -server vtctld:15999 TabletExternallyReparented {successorAlias}&quot;</span>
  <span class="p">]</span><span class="err">,</span>
</code></pre></div>
<h4 id="vttablet-configuration">VTTablet configuration</h4>

<p>Normally, you need to seed Orchestrator by giving it the addresses of
MySQL instances in each shard. If you have lots of shards, this could
be tedious or error-prone.</p>

<p>Luckily, Vitess already knows everything about all the MySQL instances
that comprise your cluster. So we provide a mechanism for tablets to
self-register with the Orchestrator API, configured by the following
vttablet parameters:</p>

<ul>
<li><strong>orc_api_url</strong>: Address of Orchestrator&#39;s HTTP API (e.g. <a href="http://host:port/api/">http://host:port/api/</a>). Leave empty to disable Orchestrator integration.</li>
<li><strong>orc_discover_interval</strong>: How often (e.g. 60s) to ping Orchestrator&#39;s HTTP API endpoint to tell it we exist. 0 means never.</li>
</ul>

<p>Not only does this relieve you from the initial seeding of addresses into
Orchestrator, it also means new instances will be discovered immediately,
and the topology will automatically repopulate even if Orchestrator&#39;s
backing store is wiped out. Note that Orchestrator will forget stale
instances after a configurable timeout.</p>

      </div>
    </div>
  </div>

</div>

    <div class="page-spacer"></div>
    <footer role="contentinfo" id="site-footer">
  <nav role="navigation" class="menu bottom-menu">
    
    <a href="https://groups.google.com/forum/#!forum/vitess" target="_blank">Contact: vitess@googlegroups.com</a>&nbsp;&nbsp;<b>·</b>&nbsp;&nbsp;
    <a href="https://groups.google.com/forum/#!forum/vitess-announce" target="_blank">Announcements</a>&nbsp;&nbsp;<b>·</b>&nbsp;&nbsp;
    &#169; 2017 <a href="/">Vitess</a> powered by <a href="https://developers.google.com/open-source/">Google Inc</a>
  </nav>
</footer>


        <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
<!--    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.2/jquery.min.js"></script>   -->
    <!-- Include all compiled plugins (below), or include individual files as needed -->
<!--
    <script src="/libs/bootstrap/js/bootstrap.min.js"></script>
-->
    <!-- Include the common site javascript -->
    <script src="/js/common.js" type="text/javascript" charset="utf-8"></script>


  </body>
</html>
